Mobile virtual desktop

ABSTRACT

A device includes a display screen a processor to render a display of a portion of a logical image on the screen. The logical image is larger than that which can be displayed on the display screen of the device. As the device is maneuvered through space, different portions of the logical image are displayed, and the user may interact with these portions as desired.

BACKGROUND

[0001] 1. Field

[0002] This disclosure relates to mobile devices, more particularly tomanaging information on mobile devices.

[0003] 2. Background

[0004] Mobile devices include such things as personal digital assistants(PDAs) like the Palm® devices, cell phones, and palmtop PCs. However,several of these devices have very small screens and cannot renderrepresentations of all of the information or applications available onthe device. The user interfaces with the information or the applicationsvia a series of inputs, such as mouse scrolls, push buttons, menuselections or a combination of these. This can be repetitious and timeconsuming.

[0005] In some instances, the graphical user interface (GUI) on mostmobile devices has inherited design rules and approaches fromconventional desktop personal computers. On most desktop PCs, however,there is enough room to see the entire desktop on the screen. Mobiledevices do not have that luxury.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The invention may be best understood by reading the disclosurewith reference to the drawings, wherein:

[0007]FIG. 1 shows an embodiment of a mobile virtual desktop system.

[0008]FIG. 2 shows an embodiment of a mobile device.

[0009]FIG. 3 shows an embodiment of a mobile virtual desktop.

[0010]FIG. 4 shows an alternative view of a mobile virtual desktop.

[0011]FIG. 5 shows an embodiment of a control gesture.

[0012]FIG. 6 shows an embodiment of an object on a mobile virtualdesktop displayed upon a mobile device.

[0013]FIG. 7 shows an embodiment of a room with active zones for amobile virtual desktop.

[0014]FIG. 8 shows an embodiment of a mobile virtual desktop softwarestack.

[0015]FIG. 9 shows an alternative embodiment of a mobile device.

[0016]FIG. 10 shows an embodiment of a game space created by two mobiledevices.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017]FIG. 1 shows an embodiment of a mobile virtual desktop system. Themobile virtual desktop (MVD) represents information upon a graphical‘desktop’ just as a conventional personal computer graphical userinterface (GUI) does. The MVD extends this model to bring a richeroperating environment to mobile, or handheld, devices. In a conventionalPC system, the logical desktop is synonymous with the physicaltwo-dimensional display screen. Desktop objects, such as applicationicons, images and text are accessed and manipulated by a physical inputdevice such as a keyboard, a mouse or a stylus pad or touch screen.Generally, in this type of environment all of the desired informationobjects in the logical desktop are rendered in the physical display. Aninformation object is an object that is selectable from the desktop,where the desktop is the interface provided to the user for display ofthese objects. Navigation occurs via a discrete input device such asthose mentioned above.

[0018] GUIs on conventional handheld devices, such as “pocket” PCs,personal digital assistants (PDAs) such as the Palm® series of devices,and cell phones, have inherited design restrictions from theconventional desktop PC. Typically, every logical page of data fits intothe physical display and a cursor icon glides over the desktop to selectinformation objects. The whole user desktop is seen on the screen, bychanging the screen shots like a deck of cards. Text is typicallyformatted to fit the physical dimensions of the display screen.

[0019] It is possible to view the handheld, or mobile, device as a newdevice rather than a reduced-size PC. Mobile devices are light and easyto grasp and move in space. Secondly, the typically small size of mobiledevices do not allow more conventional input devices to scale to humansize, such as keyboards or numerical keypads as in cell phones. Thismakes them far less useful than on a PC. Finally, many of the devicesovercome this lack of keyboards by using a stylus and touch pad, butthat requires two hands, which can be awkward.

[0020] In the mobile virtual desktop system, as shown in FIG. 1, thehandheld device becomes a ‘lens’ into a virtual desktop that is largerthan that displayed on the device. The user 12 holds the mobile device10, which has a display screen 16. The logical image 14 a is much largerthan the physical display screen 16. There may be several ‘planes’ ofthe logical image, such as 14 b, 14 c through 14 d, as will be discussedin more detail later. The MVD renders the entire logical image to avirtual viewing plane, but actually only physically renders a smallportion of that plane to display 16. As the user moves the device in thelogical information plane, information objects pass into and out ofview, with the device 10 acting like a lens into the larger logicalimage.

[0021] As the user moves the device around the virtual desktop, orlogical image, motion sensors support navigation and software tracks thedevice's location in physical space. Control semantics are associatedwith a variety of arm and hand motions that allow the user to executecommands and act upon information with the handheld device acting as amouse cursor and buttons, keyboard or other user input devices.

[0022] An example of a mobile device is shown in FIG. 2. The device 10includes the display screen 16. The user input devices may includecontrol buttons 26, or a keypad 22. The keypad 22 may further comprise anumerical keypad having keys similar to the exploded view 24, such as acell phone, or a full styled keyboard. In addition, or in lieu of theseinput devices, the device may include a stylus 20 and the display screen16 may be a touch-sensitive screen. Again, alternatively or lieu ofthese user input devices, the device may include a microphone 28, whichmay also serve as the mouthpiece of a cellular phone. These devices maynot be necessary, as will be discussed in more detail later, the usermay actually use the device itself as an input device. But MVD does notpreclude their use in conjunction with MVD control mechanisms.

[0023] An internal view of the device 10 shows that it has a positiondetermining circuit 32, such as an accelerometer or other motion sensorarray. The position determining mechanism is self-contained within thedevice and does not rely upon any external mechanisms. It allows thesystem to compute its position relative to a dynamically set homecoordinate. The position determining system also determines thedirection and nature of the movement, a processor 34 and a memory 36.There is nothing in the position determining system that precludes itfrom working with, in a complementary manner, an external positiondetermination system, such as, GPS or other radio tagging systems togain additional functionality.

[0024] The processor may be operable to render a portion of the logicalimage on the screen. The portion of the logical image being renderedwill depend on the device's position relative to the logical image. Thedevice's position may also determine the ‘active’ application or object.The motion sensor array will also serve to notify the processor when theuser performs a control gesture. The user may ‘zero’ or define a homecoordinate for the device to allow the processor to translate themotions into a current device position.

[0025] The memory 36 may store a ‘dictionary’ or other list of controlgestures and their meanings. This allows the user to move the device ina predetermined pattern or motion and have that motion translate into anexecutable command for the device, such as a mouse click or a buttonpress in a conventional PC would. The processor would access thedictionary to determine what the actions are relative to a particulargesture. However, the processor may just compute distance and directionto determine what portion of the logical image is to be displayed.

[0026] The interaction between the mobile device and the logical devicecan be seen more clearly in the examples of FIG. 3. When the user ‘turnson’ the desktop, the user may need to ‘zero’ the device. This may bedone spatially, where the device defaults to a display of the lower leftcorner of the logical image. The user then moves the device to aposition that locates that corner of the image in a comfortable positionand then presses a button to set that position as the home coordinates.Alternately, the user could pan around the logical image until a homingobject is located and the selected by a command gesture or button pressor other user input.

[0027] Once the device is homed or zeroed, the user then moves it aroundthe logical image. One example of this is shown in FIG. 3. The device 10is positioned in the logical image 14 a such that object 42 are in view.The user cannot see the other objects in the plane. The objects 40 and44 represent information objects, such as icons for applications,documents or other objects in the logical image, but not seen by theuser.

[0028] In FIG. 4, the user has moved the device relative to the objectdisplayed in FIG. 3, up and to the right. The logical image may behorizontal or vertical to the plane of the Earth. If the logical imagewere horizontal, the move would be out and to the right, rather than upand to the right. For purposes of the example, assume that the userwishes to select the object 44 shown in the display 16 of device 10 inFIG. 4.

[0029] For ease of discussion, all of the objects in the image plane notselected are eliminated from the illustration of FIG. 5. Those objectsare still in the logical image, but the user cannot see them. In orderto select the object represented by object 44, the user may centercrosshairs 46 over the object and then perform a command gesture. Oneexample of a command gesture may be for the user to move the devicesharply in the clockwise direction and then back to ‘neutral’ in thecounter clockwise direction. This gesture would have been predefined tocause the device to open or execute the object in view, referred to hereas the selected object. Alternatively, the user may perform a moreconventional user input such as a button press or provide a speechinput.

[0030] The gesture dictionary may be a configurable database of uniquemotion signatures associated with standard navigational operations, suchas selecting an object, opening an object, scrolling, increasing ordecreasing a slider, closing an application, saving a document, etc. Auser could customize the MVD operating environment by creating ormodifying a motion signature for a set of predetermined commands, or maybe able to develop the user's own commands and motions, similar todefining macros in a word processing application.

[0031] Generally, gestures may have two components, spatial directionand timing. As the device moves in space, it will have an instantaneousdirection and speed, and possibly acceleration, value. Thesemeasurements are obtained from the motion sensor array and processed todetermine location in the desktop boundaries, as well as translated intocommands for the application or object in view. The device could be‘trained’ similar to handwriting or voice recognition to recognizecertain basic gestures. In the example of FIG. 5, the gesture istranslated to open the document.

[0032]FIG. 6 shows the rendering of the object 44 on the display of thedevice. In this example, object 44 was an e-mail about a meeting. Thefirst portion of the e-mail is displayed on the screen 16. If the userwants to view the remaining portions of the email, shown by box 48, theuser moves the device down along the direction shown by the arrow. Thisoperates the device as an input device, rather than relying upon a moreconventional and possibly awkward input means such as a key or stylusinput.

[0033] Up to this point, the discussion has been as if the device isoperating in device relative mode. There are two modes, device relativeand user relative. Within user relative mode, there are also twosubcategories, user awareness mode and room awareness mode. Each ofthese will be addressed in turn. In device relative mode, the logicalimage is that as shown by logical image 14 a of FIG. 1, atwo-dimensional image of objects that are larger than can be displayedon the mobile device.

[0034] However, as can be seen in FIG. 1, the logical image may haveseveral different planes, each one a predefined distance from theothers. In the example of FIG. 1, the different image planes are assumedto be a different distance away from the user and the logical image isoriented vertically. If the logical image were oriented horizontally,the different image planes would be located at different distances fromthe Earth or a bottom plane, stacked in front of the user. The use of athree-dimensional logical image occurs in user relative mode.

[0035] With a three dimensional image, the Z-axis becomes a navigationalpart of the interaction, instead of just an interpretation of a controlgesture. The movements along the Z-axis can be tracked in differentways, two examples are in user awareness mode and room awareness mode.In user awareness mode, the different planes of information are definedas being relative to the front of the user's body. The user will home orzero the device as the front plane of the logical image, for example.This may be 6-8 inches from the front of the user's body. As the usernavigates the logical image, a movement along the Z-axis another 6-8inches away from that plane may cause another plane to come into view.The user does not necessarily need to remember the exact depths set bythe different image planes, as the user can scan around with the deviceuntil the object being displayed changes to one on the next plane of theimage.

[0036] In a further adaptation of this approach, the device couldoperate in a ‘room awareness’ mode. Any physical room could be mappedout to contain certain task-specific zones, referred to here as hotzones or active zones. Because the MVD handheld has the capability torecognize when it is in a particular hot-zone, it can be configured tobehave in a manner that is specific to the hot-zone. For example, thisbehavior might be pre-set into the handheld by software installed by acompany's information technology administrator per the company'ssecurity or safety policy. The zones are areas of a room where work canbe done with a specific number and type of applications. For example, aconference table may be defined as a ‘meeting hot zone’ where only thatinformation on the logical image that is ‘public’ may be accessible byother users and their devices. A port 30, shown in FIG. 2, mayfacilitate sharing of information.

[0037] The hot zones may cause the device applications and objects to bebound, or it may activate certain applications. For example, assume auser enters the room shown in FIG. 7 with the device 10. In the room area printer 56, a PC 58, and a conference table 52 with a teleconferencephone 54 upon it. The room beacon 60 allows an MVD device toautomatically home itself at the door. From then on, it would use itsself-contained position determining mechanism to locate itself withinthe room.

[0038] The room itself has an effect upon which elements of the logicaldesktop get displayed in the MVD desktop such as the granted level ofresource access and control. For example, consider a ‘restricted’employee who is not granted access to company confidential information.When the restricted MVD user sits at the conference table at the centerof the room, the user's MVD will have calculated within the room thatthe user is at the center of the room where the table is. While in thishot zone, no private company information will be accessible.

[0039] In order to facilitate this type of room awareness, the user maymap different rooms and save them in several different profiles such asConference Room 1, Auditorium 1, etc. The MVD device would be programmedwith the coordinates of the room's hot zones. Whenever the user entersthat room, the user would zero the MVD device's coordinates relative tothe room coordinates of hot zones. This allows the device to alter thelogical image presented depending upon the device location. This processmay be further facilitated by the use of a beaconing device such as thatshown as 60 in FIG. 7. This would allow the device to determine itsposition relative to the incoming signal, and thereby establish its homeposition.

[0040] The above discussion is intended merely as an example. Thelogical image could be constructed in whatever manner the systemdesigner and the device capabilities allow. For example, the user couldnavigate an image similar to a representation of an actual desktop,almost as if an invisible holographic image of the user's actual desktopin the user's office were in existence.

[0041] An aspect of the MVD that allows the functionalities and featuresdiscussed is a software stack that includes ‘middleware.’ The middlewareis software that translates between the physical movement and positionof the device and the various graphical user interface (GUI) applicationprogramming interfaces (APIs) of the various applications and objectsbeing operated upon. An example of such a stack is shown in FIG. 8.

[0042] The operating system 84 of the device is supported by theoperating system GUI APIs 82, which perform the function calls to theoperating system 84. The middleware layer 76 generates the functioncalls. The middleware manages the gestures dictionary 80, and provideslogic to associate gestures with the computing actions. The middlewarealso provides support to the user interface that allows the user toconfigure the gestures dictionary and to interact with the behaviors ofthe logical image.

[0043] The middleware manages the current state of the MVD desktoplogical image and the processes. These include the locationdetermination manager 79 and the gesture control manager 79. Thesetranslate the gestures and movements into state transitions and functioncalls for the operating system GUI APIs that allows the device toexecute operating system commands. In addition, the middleware performsthe interface tasks between the motion sensor hardware and the motionsensor drivers that allow the middleware to determine the position andmotions of the device.

[0044] In some embodiments, the processes of the invention may beincluded in an article of machine-readable code that, when executedcauses the machine to execute the various processes of the invention.

[0045] The MVD applications and use is not limited to any particularconfiguration of the device, nor is it restricted to the personaldigital assistant functionalities commonly used with mobile devices. InFIG. 9, for example, an alternative configuration is shown for themobile device. The mobile device 10 is mounted on a watchband 80. Thedevice may rely upon gestures only to provide user interaction with theapplications and data, or may include some control buttons or othertypes of inputs, such as speech.

[0046] In this embodiment, the display 16 may function as a wristwatchdisplay until the user activates the MVD, although no watchfunctionality is necessary. At that point, the display may opticallyswitch to performing like a microdisplay. A microdisplay is a displaythat has been optically configured to project images directly to theuser's eye and typically requires that the display be held very close tothe eyes. While holding the watch and device up close to the eye may beuncomfortable, it would allow the user to take advantage of thefunctions of the mobile device and the capabilities of the MVD.

[0047] However, it is also possible that the use of gestures in such aconstrained configuration makes the translation of the gesturesimprecise. It is also possible that the display 16 may function just asa watch face and possibly as an initial screen. The mobile device 10would then transmit the information to be displayed through the port 30to a pair of wearable lenses, on a separate eyepiece such as eyeglasses,82. The receiver 84 would then receive the display data and render it onone or both of the lenses of the eyepiece 82.

[0048] The MVD is also easily extensible beyond the PDA or cell phoneapplications. For example, a first user may launch a game application ona first mobile device 10, while a second user then launches a gameapplication on a second mobile device 90. The two devices would alsoestablish a mutual game space 92 in a peer-to-peer link. The game spacemay be established by zeroing and defining regions for each device andthen communicated those coordinates to the other device. In this manner,the users may ‘share’ a MVD space that differs greatly from a shareddesktop.

[0049] Thus, although there has been described to this point aparticular embodiment for a method and apparatus for a mobile virtualdesktop, it is not intended that such specific references be consideredas limitations upon the scope of this invention except in-so-far as setforth in the following claims.

What is claimed is:
 1. A device, comprising: a display screen; and aprocessor to render a display of a portion of a logical image on thescreen.
 2. The device of claim 1, wherein the device further comprises amemory to store a dictionary of gestures.
 3. The device of claim 1,wherein the device further comprises a user input device.
 4. The deviceof claim 3, wherein the user input device further comprises one of thegroup comprised of: a microphone, a stylus, a keyboard, numericalkeypad, and a mouse.
 5. The device of claim 1, wherein the device isoperable to act as a user input device.
 6. The device of claim 1,wherein the logical image is two-dimensional.
 7. The device of claim 1,wherein the logical image is three-dimensional.
 8. The device of claim1, wherein the device is mounted on a watchband.
 9. The device of claim1, wherein the device further comprises a separate eyepiece display. 10.A device, comprising: a position determining circuit; ; and a processorto interpret movement of the device based upon inputs received from theaccelerometers and to determine an active application based upon themovement.
 11. The device of claim 10, wherein the device furthercomprises a user interface to allow the user to define a home coordinatefor the device.
 12. The device of claim 10, wherein the positiondetermining circuit further comprises a self-contained mechanism. 13.The device of claim 10 wherein the device further comprises a memory tostore gestures and corresponding actions based upon those gestures. 14.The device of claim 10, wherein the device operates in a mode selectedfrom the group comprises of: device relative, user awareness, and roomawareness.
 15. The device of claim 10, wherein the device operates inroom awareness mode only has access to objects specified as relevant fora particular room
 16. The device of claim 15, wherein the objectsspecified as relevant for a particular room are in accordance with acompany security policy.
 17. The device of claim 10, wherein the deviceis in communication with a second device and the two devices share acommon game space.
 18. The device of claim 10, wherein the devicefurther comprises a port to allow communication with other devices. 19.A method of translating information, the method comprising: determininga current position of a mobile device in a mobile virtual desktop;interpreting a gesture made with the mobile device on the mobile virtualdesktop; and translating the gesture from the mobile virtual desktop toat least one command for an operating system on the mobile device. 20.The method of claim 19, wherein the method further comprises receiving auser input indicating a home position for the device.
 21. The method ofclaim 19, wherein determining a current position further comprisesdetermining a current position on two axes of movement.
 22. The methodof claim 19, wherein determining a current position further comprisesdetermining a current position on three axes of movement.
 23. The methodof claim 19, wherein determining a current position further comprisesderiving movement magnitude and direction from a home position.
 24. Themethod of claim 19, wherein interpreting a gesture further comprisesdetermining a movement and timing from a gesture made with the mobiledevice.
 25. The method of claim 19, wherein translating the gesturefurther comprises accessing a database using movement and timinginformation as indexes, and determining a command from the movement andtiming information.
 26. An article including machine-readable code that,when executed, causes the machine to: determine a current position of amobile device in a mobile virtual desktop; interpret a gesture made witha mobile device on the mobile virtual desktop; and translate the gesturefrom the mobile virtual desktop to at least one command for an operatingsystem on the mobile device.
 27. The article of claim 26, wherein thecode that, when executed, causes the machine to determine the positionof a mobile device further comprises determining movement magnitude anddirection from a home position.
 28. The article of claim 26, wherein thearticle further includes code that, when executed, cause the machine toreceive a user input indicating a home position.
 29. A method oftranslating a gesture, the method comprising: determining an initialposition of a device; receiving at least one signal indicating amovement of the device; and executing a command corresponding to themovement signal.
 30. The method of claim 29, wherein the method furthercomprises receiving a user input indicating a home position of thedevice.
 31. The method of claim 29, wherein the signal indicating amovement of the device further comprises a spatial direction signal anda timing signal.
 32. The method of claim 29, wherein executing a commandcorresponding to the movement signal further comprises executing acommand within an application identified by the initial position of thedevice.
 33. An article including machine-readable code that, whenexecuted, causes the machine to: determine an initial position of adevice; receive at least one signal indicating a movement of the device;and execute a command corresponding to the movement signals.
 34. Thearticle of claim 33, wherein the article further includes code that,when executed, cause the machine to receive a user input indicating ahome position of the device.
 35. The article of claim 33, wherein thesignal indicating a movement of the device further comprises a spatialdirection signal and a timing signal.